H.264/MPEG-4 AVC is a standard for digital video compression jointly developed by ITU-T’s Video Coding Experts Group (VCEG) and ISO/IEC’s Moving Picture Experts Group (MPEG). Next to efficient coding algorithms, the H.264/AVC specifications define a set of error resilience tools. The most striking one is Flexible Macroblock Ordering or FMO.
Contents |
One of the new characteristics of the H.264/AVC standard is the possibility of dividing an image in regions called slice groups. Each slice group can also be divided in several slices; therefore, we need to define the word slice as a sequence of macroblocks that belong to the same slice group. These macroblocks are processed in a scan order (left to right and top to bottom). A slice can be decoded independently.
FMO consists of deciding to which slice each macroblock of the image belongs. Each macroblock can be assigned freely to a slice group using an MBAmap (MacroBlock Allocation map). The MBAmap consists of an identification number for each macroblock of the image that specifies to which slice group that macroblock belongs. The number of slice groups is limited to 8 for each picture to prevent complex allocation schemes.
If we deactivate the FMO, the images will be composed of a single slice group with the macroblocks in a scan order. The use of FMO is totally compatible with any type of inter-frame prediction.
With this technique, we can correct errors easily by exploiting the spatial redundancy of the images. It is a good idea to choose the slice groups in a way that no macroblock and its neighbors belong to the same group. Therefore, if a slice is lost during transmission, it is possible to reconstruct the lost blocks with the information of the neighboring blocks. We have to consider the transmission characteristics of these slices: each slice is transmitted independently in separate units called packets. Each packet contains in its own header the information to decode itself without any other packet’s information (if the images used as reference are the same in the encoder and the decoder side). The use of FMO together with advanced error resilience tools can keep the visual quality even with a packet loss rate of 10%.
When using FMO, the image can be divided in different scan patterns of the macroblocks. There are seven FMO map types, referred to as Type 0 through Type 6. Type 6 is the most general one and allows full flexibility to the user. The others use specific pattern rules. These patterns can be exploited to represent the MBAmap more efficiently:
Note: Each color represents a slice group
In this section we describe some of the possible applications of FMO in H.264/AVC video coding:
▪ The FMO Type 1 can also be used in transmission environments with a high packet loss rate (example seen in Experimental Results).
▪ The FMO Type 2 can also be very useful. Let’s imagine we want to transmit a news bulletin video with a low bit rate. If we encode the whole image with the bit rate distributed equally for all the macroblocks we'll get some bad results. Humans are particularly sensitive to image errors in faces, so we can mark the region of the newsreader’s face so it's encoded with more bits. Therefore, the background of the set will be encoded with less bits and the newsreader with a lot more bits (maintaining the total bit rate as we had before). This way, the subjective visual quality will be enhanced significantly.
To do these experiments we have used reference software of the H.264/AVC standard [1]. In the images below we can see two frames of the Stefan.yuv sequence, the first one without using FMO and the second one using FMO Type 1:
We can clearly see in the bottom-right corner of the image that some packets were lost during transmission and the macroblocks of the image couldn’t be decoded. But when we use FMO Type 1, we have two slice groups dividing the image like a chess board. Therefore, when a packet of the slice group 0 is lost, we can recover that part of the image interpolating the neighboring macroblocks that have not been lost because they belong to slice group 1.
This overview of FMO and the experimental results appearing in this article clearly show that FMO is a very useful error resilience tool. Its use can enhance visual quality, both subjective and objective (SNR or MSE), when the packets are being transmitted in an environment with a certain packet loss rate. Even though we’ve been exposing the advantages of FMO, we also have to consider its main disadvantage: we have to send the information of which FMO we’re using in the header of the packet. That means we have a significant increase in the number of bits we have to send. Therefore, the use of FMO is only recommended when we know that the transmission environment is likely to have a rather high packet loss rate.